Existing pre-training methods for extractive Question Answering (QA) generate cloze-like queries different from natural questions in syntax structure, which could overfit pre-trained models to simple keyword matching. In order to address this problem, we propose a novel Momentum Contrastive pRe-training fOr queStion anSwering (MCROSS) method for extractive QA. Specifically, MCROSS introduces a momentum contrastive learning framework to align the answer probability between cloze-like and natural query-passage sample pairs. Hence, the pre-trained models can better transfer the knowledge learned in cloze-like samples to answering natural questions. Experimental results on three benchmarking QA datasets show that our method achieves noticeable improvement compared with all baselines in both supervised and zero-shot scenarios.

Hyperbolic space is emerging as a promising learning space for representation learning, owning to its exponential growth volume. Compared with the flat Euclidean space, the curved hyperbolic space is far more ambient and embeddable, particularly for datasets with implicit tree-like architectures, such as hierarchies and power-law distributions. On the other hand, the structure of a real-world network is usually intricate, with some regions being tree-like, some being flat, and others being circular. Directly embedding heterogeneous structural networks into a homogeneous embedding space unavoidably brings inductive biases and distortions. Inspiringly, the discrete curvature can well describe the local structure of a node and its surroundings, which motivates us to investigate the information conveyed by the network topology explicitly in improving geometric learning. To this end, we explore the properties of the local discrete curvature of graph topology and the continuous global curvature of embedding space. Besides, a Hyperbolic Curvature-aware Graph Neural Network, HCGNN, is further proposed. In particular, HCGNN utilizes the discrete curvature to lead message passing of the surroundings and adaptively adjust the continuous curvature simultaneously. Extensive experiments on node classification and link prediction tasks show that the proposed method outperforms various competitive models by a large margin in both high and low hyperbolic graph data. Case studies further illustrate the efficacy of discrete curvature in finding local clusters and alleviating the distortion caused by hyperbolic geometry.

Although augmentations (e.g., perturbation of graph edges, image crops) boost the efficiency of Contrastive Learning (CL), feature level augmentation is another plausible, complementary yet not well researched strategy. Thus, we present a novel spectral feature argumentation for contrastive learning on graphs (and images). To this end, for each data view, we estimate a low-rank approximation per feature map and subtract that approximation from the map to obtain its complement. This is achieved by the proposed herein incomplete power iteration, a non-standard power iteration regime which enjoys two valuable byproducts (under mere one or two iterations): (i) it partially balances spectrum of the feature map, and (ii) it injects the noise into rebalanced singular values of the feature map (spectral augmentation). For two views, we align these rebalanced feature maps as such an improved alignment step can focus more on less dominant singular values of matrices of both views, whereas the spectral augmentation does not affect the spectral angle alignment (singular vectors are not perturbed). We derive the analytical form for: (i) the incomplete power iteration to capture its spectrum-balancing effect, and (ii) the variance of singular values augmented implicitly by the noise. We also show that the spectral augmentation improves the generalization bound. Experiments on graph/image datasets show that our spectral feature augmentation outperforms baselines, and is complementary with other augmentation strategies and compatible with various contrastive losses.

Information Extraction (IE) aims to extract structured information from heterogeneous sources. IE from natural language texts include sub-tasks such as Named Entity Recognition (NER), Relation Extraction (RE), and Event Extraction (EE). Most IE systems require comprehensive understandings of sentence structure, implied semantics, and domain knowledge to perform well; thus, IE tasks always need adequate external resources and annotations. However, it takes time and effort to obtain more human annotations. Low-Resource Information Extraction (LRIE) strives to use unsupervised data, reducing the required resources and human annotation. In practice, existing systems either utilize self-training schemes to generate pseudo labels that will cause the gradual drift problem, or leverage consistency regularization methods which inevitably possess confirmation bias. To alleviate confirmation bias due to the lack of feedback loops in existing LRIE learning paradigms, we develop a Gradient Imitation Reinforcement Learning (GIRL) method to encourage pseudo-labeled data to imitate the gradient descent direction on labeled data, which can force pseudo-labeled data to achieve better optimization capabilities similar to labeled data. Based on how well the pseudo-labeled data imitates the instructive gradient descent direction obtained from labeled data, we design a reward to quantify the imitation process and bootstrap the optimization capability of pseudo-labeled data through trial and error. In addition to learning paradigms, GIRL is not limited to specific sub-tasks, and we leverage GIRL to solve all IE sub-tasks (named entity recognition, relation extraction, and event extraction) in low-resource settings (semi-supervised IE and few-shot IE).

Graph-structured data are widespread in real-world applications, such as social networks, recommender systems, knowledge graphs, chemical molecules etc. Despite the success of Euclidean space for graph-related learning tasks, its ability to model complex patterns is essentially constrained by its polynomially growing capacity. Recently, hyperbolic spaces have emerged as a promising alternative for processing graph data with tree-like structure or power-law distribution, owing to the exponential growth property. Different from Euclidean space, which expands polynomially, the hyperbolic space grows exponentially which makes it gains natural advantages in abstracting tree-like or scale-free graphs with hierarchical organizations. In this tutorial, we aim to give an introduction to this emerging field of graph representation learning with the express purpose of being accessible to all audiences. We first give a brief introduction to graph representation learning as well as some preliminary Riemannian and hyperbolic geometry. We then comprehensively revisit the hyperbolic embedding techniques, including hyperbolic shallow models and hyperbolic neural networks. In addition, we introduce the technical details of the current hyperbolic graph neural networks by unifying them into a general framework and summarizing the variants of each component. Moreover, we further introduce a series of related applications in a variety of fields. In the last part, we discuss several advanced topics about hyperbolic geometry for graph representation learning, which potentially serve as guidelines for further flourishing the non-Euclidean graph learning community.

考虑到用户项目网络中幂律分布的流行率，双曲线空间最近引起了人们的关注，并在推荐系统中获得了令人印象深刻的性能。双曲线推荐的优点在于，其指数增加的能力非常适合描述幂律分布式用户项目网络，而欧几里得等效的不足。尽管如此，尚不清楚双曲模型可以有效地推荐哪些项目，哪些项目不能。为了解决上述问题，我们采用最基本的建议技术，将协作过滤作为一种媒介，以研究双曲线和欧几里得建议模型的行为。结果表明，（1）尾部在双曲线空间中比在欧几里得空间中更重点，但是仍然有足够的改进空间。 （2）头部物品在双曲线空间中受到适度的关注，这可以大大改善； （3）尽管如此，双曲线模型比欧几里得模型表现出更具竞争力的性能。在上述观察结果的驱动下，我们设计了一种新颖的学习方法，称为双曲线信息合作过滤（HICF），旨在弥补头部项目的建议有效性，同时提高尾部项目的性能。主要的想法是调整双曲线的排名学习，使其拉力和推动程序几何了解，并为学习头和尾部的学习提供信息指导。广泛的实验备份了分析结果，还显示了该方法的有效性。这项工作对于个性化的建议很有价值，因为它揭示了双曲线空间有助于建模尾部项目，这通常代表用户定制的偏好或新产品。

RNA结构的确定和预测可以促进靶向RNA的药物开发和可用的共性元素设计。但是，由于RNA的固有结构灵活性，所有三种主流结构测定方法（X射线晶体学，NMR和Cryo-EM）在解决RNA结构时会遇到挑战，这导致已解决的RNA结构的稀缺性。计算预测方法作为实验技术的补充。但是，\ textit {de从头}的方法都不基于深度学习，因为可用的结构太少。取而代之的是，他们中的大多数采用了耗时的采样策略，而且它们的性能似乎达到了高原。在这项工作中，我们开发了第一种端到端的深度学习方法E2FOLD-3D，以准确执行\ textit {de de novo} RNA结构预测。提出了几个新的组件来克服数据稀缺性，例如完全不同的端到端管道，二级结构辅助自我鉴定和参数有效的骨干配方。此类设计在独立的，非重叠的RNA拼图测试数据集上进行了验证，并达到平均sub-4 \ aa {}根平方偏差，与最先进的方法相比，它表现出了优越的性能。有趣的是，它在预测RNA复杂结构时也可以取得令人鼓舞的结果，这是先前系统无法完成的壮举。当E2FOLD-3D与实验技术耦合时，RNA结构预测场可以大大提高。

概念相关性估计（CRE）任务是确定两个给定的概念是否相关。尽管可以轻松适应此任务的语义文本相似性（STS）任务的现有方法，但CRE任务具有一些独特的属性，可以利用这些属性来扩大数据集以解决其数据稀缺问题。在本文中，我们构造了一个名为CycreteGraph（概念相关性估计图）的图，以利用CRE属性。对于从混凝土图中采样的新概念对，我们添加了一个额外的步骤，以基于简单但有效的质量阈值来滤除低质量的新概念对。我们将ConcreteGraph数据扩展应用于三个基于变压器的模型以显示其功效。详细的消融研究用于质量阈值进一步表明，即使有限的高质量数据也比大量未替代数据更有益。本文是第一个在数据集上使用的文章，而建议的具体图可以提高变压器的准确性超过2％。在CNSE和CNSS数据集上，所有三个变压器都借助ConcreteGraph，均可超越当前最先进的方法，概念交互图（CIG）。

图形神经网络（GNN）由于从图形结构数据中学习表示能力而引起了很多关注。尽管GNN在许多域中成功地应用了，但GNN的优化程度较低，并且在节点分类的性能很大程度上受到了长尾节点学位分布的影响。本文着重于通过归一化提高GNN的性能。详细说明，通过研究图中的节点度的长尾巴分布，我们提出了一种新颖的GNN归一化方法，该方法称为RESNORM（\ textbf {res}将长尾巴分布纳入正常分布，通过\ textbf {norm} alization）。 RESNOR的$比例$操作重塑节点标准偏差（NSTD）分布，以提高尾部节点的准确性（\ textit {i}。\ textit {e}。，低度节点）。我们提供了理论解释和经验证据，以理解上述$ scale $的机制。除了长期的分销问题外，过度光滑也是困扰社区的基本问题。为此，我们分析了标准偏移的行为，并证明了标准移位是重量矩阵上的预处理，从而增加了过度平滑的风险。考虑到过度光滑的问题，我们为Resnorm设计了一个$ Shift $操作，以低成本的方式模拟了特定于学位的参数策略。广泛的实验验证了重新分类对几个节点分类基准数据集的有效性。

为了减轻传统推荐系统（RSS）的数据稀疏和冷启动问题，将知识图（KGS）纳入补充辅助信息，最近引起了相当大的关注。然而，简单地整合了基于KG的RS模型的KGS，这不一定是提高推荐性能的保证，甚至可能削弱整体模型能力。这是因为这些KG的构建与历史用户项相互作用的集合无关;因此，这些KG的信息可能并不总是有助于推荐给所有用户。在本文中，我们提出了具有个性化推荐的协作指导的细心知识意识的图表卷积网络（CG-KGR）。 CG-KGR是一种新颖的知识意识推荐模型，通过我们提出的协作指导机制，可以实现高度和相干的KG和用户项目交互的学习。具体而言，CG-KGR首先封装与交互式信息摘要的历史相互作用。然后CG-kgr利用它作为提取kgs的信息的指导，最终提供更精确的个性化推荐。我们在两个推荐任务中对四个现实数据集进行了广泛的实验，即TOP-K推荐和点击率（CTR）预测。实验结果表明，CG-KGR模型在Top-K推荐的召回度量方面，最近最初的最先进模型明显优于1.4-27.0％。